The present invention relates to an image pick-up apparatus that is capable of providing images with high resolution, and more particularly, concerns an image pick-up apparatus which picks up a plurality of images by relatively shifting imaging light from a subject, and combines these images into an image with an enhanced resolution.
Recently, still-image pick-up apparatuses, so-called electronic still cameras, have been put into practical use as image pick-up apparatuses for picking up still images. Further, home-use video cameras have been put into practical use as motion-image pick-up apparatuses for picking up motion images. In these image pick-up apparatuses, a charge-coupled device is used as a means for picking up an image by receiving imaging light from a subject.
The charge-coupled device is a so-called two-dimensional CCD image sensor. In the charge-coupled device, a plurality of light-receiving regions are arranged in a matrix format on an image-forming surface that is a two-dimensional plane.
The image pick-up apparatus converges imaging light from a subject onto the image-forming surface of the charge-coupled device, and allows the light-receiving regions to receive the light. The imaging light is photoelectrically transferred to an electric signal indicating the quantity of light receipt within the light-receiving region, and then recorded in a recording medium as an image signal. This image signal, when visually displayed alone on a display device, forms a still image. Further, these image signals, when visually displayed successively in the order in which they were picked up form motion images.
In the image pick-up apparatus of this type, an image that has been picked up consists of pixels that correspond to light-receiving regions of the charge-coupled devices. In other words, the operation of each charge-coupled device is the same as sampling the quantity of imaging light that spatially varies in succession by using a spatial sampling frequency. The spatial sampling frequency is given as an inverse number to the array period of the pixels. Therefore, the change in light quantity of imaging light is smoothed for each pixel. Consequently, the higher the number of pixels, the more the resolution of an image is improved.
As one of the methods for improving the resolution of an image, CCD image pick-up apparatuses using an image shift have been proposed. The image shift is a technique for shifting a light-receiving position of imaging light that is directed to the charge-coupled device. In the CCD image pick-up apparatus using the image shift, a plurality of image pick-up processes are carried out while shifting the light-receiving position of imaging light from a subject on the image-forming surface. The images, picked up in this manner, are superimposed so that the light-receiving positions of the image are coincident with one another, thereby forming an output image.
Japanese Laid-Open Patent Publication No. 284980/1988 (Tokukaishou 63-284980) discloses one of such CCD image pick-up apparatuses using the image-shift system. In this CCD image pick-up apparatus, a parallel flat plate, which transmits light, is interpolated between a light-converging lens for converging light from a subject and a charge-coupled device. The parallel flat plate is aligned in either of two states, that is, the first state in which it is aligned perpendicular to the light axis and the second state in which it is inclined in an diagonal direction of 45 degrees with respect to the horizontal and vertical directions of visual field. When the parallel flat plate is aligned in the first state, the charge-coupled device picks up a first image, and thereafter, when the parallel flat plate is aligned in the second state, it picks up a second image.
FIG. 32 is a drawing that shows a pixel array equivalent to an output image. This output image forms a monochrome image. The light-receiving regions of the charge-coupled device are arranged in a matrix format with a horizontal array period PH and a vertical array period PV. Here, it is supposed that the first image and the second image have been picked up while shifting light from a subject in a diagonal direction by xc2xd pixel from each other. In this case, in the output image formed by combining the two sheets of images, the pixels are arranged with a horizontal array period of (PH/2) and a vertical array period of (PV/2). In other words, the number of pixels is increased fourfold in the entire image.
In FIG. 32, pixels s1 represent actual pixels whose pixel data has been obtained from the first original image that has been picked up in the first state. Further, pixels s2 represent actual pixels whose pixel data is obtained from the second original image that has been picked up in the second state. In FIG. 32, these actual pixels are indicated by hatched regions. Thus, in the output image, the actual pixels whose pixel data has been obtained are arranged in a diced pattern.
Each of the pixels that have no pixel data (known as virtual pixels) are adjacent to two actual pixels in each array direction. The pixel data of these virtual pixels can be obtained by, for example, interpolating the average value of the pixel data of the adjacent four actual pixels. In this manner, a conventional CCD image pick-up apparatus can obtain a high-resolution image consisting of pixels the number of which is four times as many as the number of the light-receiving regions of the charge-coupled device.
In the above-mentioned CCD image pick-up apparatus, image signals corresponding to one sheet of an output image are generated from the two original images that have been successively picked up through the image shifting process. Accordingly, in this apparatus, it is desirable to have equal exposing time upon picking up the two original images so as not to cause a difference in light quantity between the two original images. However, even in the case of equal exposing time, a difference in light quantity may occur between the two original images due to flickers of a fluorescent lamp or other reasons.
If there is a difference in light quantity between the two sheets of original images, a diced pattern, which is originally not supposed to appear in the imaging light, tends to appear in the output image, resulting in degradation in the image quality. FIG. 33 shows an example of the diced pattern that appears even when an image of a flat blank pattern is picked up. In FIG. 33, figures given in the respective pixels represent values of the pixel data in the corresponding pixels, and it is defined that pixel s1 in the first image has pixel data of 100, that is, the light quantity. Further, pixel s2 of the second image, which is originally supposed to have pixel data of 100, has a reduced light quantity due to the above-mentioned phenomenon so that it merely has pixel data of 90. In this case, if the image data of a virtual pixel is found by carrying out an average interpolation on the pixel data of the adjacent four actual pixels, the pixel data is calculated as 95. In this manner, if there is a difference in light quantity between the two sheets of images, a diced pattern appears as shown in FIG. 33, instead of a blank pattern that is originally supposed to be obtained from pixel data of 100 in all the pixels.
In order to solve the above-mentioned problem, the applicant of the present application has proposed several methods for correcting light-quantity differences in an image pick up apparatus having a light-quantity-difference correcting means that was previously filed in Japan, that is, xe2x80x9cImage pick-up apparatusxe2x80x9d (Japanese Patent Application No. 267552/1996 (Tokuganhei 8-267552)). The methods for correcting light-quantity differences, proposed by the applicant of the present application, will be described below.
First, referring to FIGS. 34 through 36, the first light-quantity-difference correction method (light-quantity-difference correction method (I)) will be discussed as follows: FIG. 34 shows a histogram of one of the blocks obtained by dividing a first original image into predetermined blocks, and FIG. 35 shows a histogram of one of the blocks obtained by dividing a second original image into predetermined blocks.
The light-quantity-difference correcting means corrects the light-quantity difference between two screens by combining the minimum value, the average value, and the maximum value among values (which take, for example, integral numbers ranging from not less than 0 to not more than 255 in the case of 8-bit recording) of the pixel data of pixels corresponding to each block of the two original images by using the following methods:
Here, it is supposed that the minimum value, the average value and the maximum value of pixel data of pixels corresponding to the first image are xcex1, xcex2 and xcex3 respectively and that the minimum value, the average value and the maximum value of pixel data of pixels corresponding to the second image are xcex4, xcex5and xcex6respectively.
Then, as shown in FIG. 36, it is supposed that the values of pixel data of the pixels corresponding to the second image before a light-quantity-difference correction are plotted on the axis of the abscissa and the values of pixel data of the pixels corresponding to the second image after the light-quantity-difference correction are plotted on the axis of the ordinate. Here, the minimum value xcex1, the average value xcex2 and the maximum value xcex4 of the pixel data corresponding to the first image are plotted on the axis of the ordinate so that point xcex7 (xcex4xcex1), point xcex8 (xcex5, xcex2) and point "igr" (xcex3,xcex6) are obtained. Further, the above-mentioned points xcex7, xcex8 and "igr" are connected by a straight line. Among the pixel data of the pixels of the second image, with regard to pixel data except for the minimum value xcex4, the average value xcex5 and the maximum value xcex3, values of the pixel data after the correction can be obtained by carrying out a projection on the axis of the ordinate with respect to the straight line made by connecting xcex7, xcex8 and "igr".
With this arrangement, the predetermined pixel values (that is, the minimum value, the average value and the maximum value) of pixel data of the pixels of the second image are respectively converted so as to coincide with predetermined values of pixel data of the first image by the light-quantity-difference correcting means.
Thus, it becomes possible to correct the light-quantity difference in imaging light between the two sheets of original images, and consequently to prevent degradation in the image quality of an output image due to the above-mentioned light-quantity difference.
Next, the second light-quantity-difference correction method (light-quantity-difference correction method (II)), which carries out a light-quantity-difference correction by finding the moving average, will be discussed as follows.
In this light-quantity-difference correction method (II), with respect to a certain pixel of the second image that is to be subjected to a light-quantity-difference correction, the moving average is found at this pixel position for each of the two sheets of original images, and the difference between the moving averages, that is, the light-quantity difference, is found. Then, the light-quantity difference is added to the pixel data of the corresponding pixel of the second image. This process is carried out on each pixel of the second image so as to perform the light-quantity-difference correction.
Moreover, in the third light-quantity-difference correction method (light-quantity-difference correction method (III)), a sensor, which detects the quantity of light for picking up each image, is separately installed, and a light-quantity-difference correction is carried out based on the values of the light quantity of each original image obtained from the sensor output.
Here, in light-quantity-difference correction methods (I) and (II) proposed by the applicant of the present application, the light-quantity-difference correction is carried out without causing any degradation in the image quality with respect to the blank pattern shown in FIG. 33 or patterns merely having gradual light-quantity changes.
However, the problem with light-quantity-difference correction method (I) is that block borders tend to appear in the output image (similar to the known block distortion appearing in image-compressing techniques) since the light-quantity-difference correction is carried out for each block. Moreover, since the light-quantity-difference correction is carried out based on the three light quantities, that is, the minimum value, the average value and the maximum value, errors in these three parameters tend to give serious adverse effects. For example, if the maximum value of the second original image signal becomes greater than the original light quantity due to noise, etc., the second original image signal, after having the light-quantity-difference correction, provides an image darker than that provided by the first original image signal, thereby leaving the diced pattern without being erased.
Moreover, the problem with light-quantity-difference correction method (II) is that false edges tend to appear at edge portions, causing double edges or an insufficient light-quantity-difference correction in the periphery of edges.
Furthermore, in light-quantity-difference correction method (III) having the separately-installed sensor for detecting light quantity, the light-quantity correction is carried out over the entire original image. For this reason, if the light-quantity difference varies depending on regions, such as a light-axis portion and peripheral portions, of the original image, it is not possible to carry out the light-quantity-difference correction properly. In particular, in the case when exposing time is controlled by a mechanical shutter, the exposing time for the vicinity of the light axis becomes longer than that for the peripheral portions, thereby causing degradation in the image quality.
The first objective of the present invention is to provide an image pick-up apparatus which, even in the case when there is a light-quantity difference between two original images or in the case when portions of the images have different light quantities, can prevent degradation of the image quality of an output image by properly correcting the light-quantity difference.
In order to achieve the above-mentioned objective, the image pick-up apparatus of the present invention is provided with: an image pick-up section for picking up imaging light from a subject and forming image data; an image-shift mechanism for shifting the image pick-up section to a plurality of relative positions with respect to the imaging light so as to allow the image pick-up section to form the image data at the relative positions; an LPF, constituted by a filter, for eliminating a spatial frequency component that is generated due to a light-quantity difference between a plurality of pieces of image data that the image pick-up section has formed by using the image-shift mechanism; and a memory control section for combining the plurality of the image data.
In the above-mentioned image pick-up apparatus, the LPF, which serves as a filter for correcting light-quantity differences occurring between a plurality of original image signals that have been generated by an image-shift operation made by the image-shift mechanism, is installed; therefore, it is possible to suppress striped patterns occurring due to the light-quantity differences, and consequently to improve the image quality.
Moreover, after combining the plurality of the original images, the filtering by the LPF is carried out prior to an interpolating process; thus, it becomes possible to reduce the amount of processes concerning the light-quantity correction (filtering), and consequently to allow high-speed processing.
Furthermore, by carrying out the filtering of the LPF after the interpolating process of the combined image, it becomes possible to suppress striped patterns occurring due to the interpolating process simultaneously with the suppression of the striped patterns occurring due to the light-quantity differences, and consequently to improve the image quality. It also becomes possible to correct edges in the horizontal or vertical direction accurately.